Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pixel grid dimensions (40818x1) error fixed. #109

Merged
merged 6 commits into from
Dec 19, 2023

Conversation

dabhicusp
Copy link
Collaborator

@dabhicusp dabhicusp commented Nov 28, 2023

Fixed: #85.

The current code gives an error(Pixel grid dimensions (40818x1) must be less than or equal to 32768) when creating an lat, lon array for the image_to_array conversion, for certain lengthy images. To address this, I've made updates to the code. Now, when generating an lat, lon array for certain lengthy images, I break it down into smaller array based on Users preferred size or default size. This adjustment prevents the above error.

@dabhicusp dabhicusp marked this pull request as ready for review November 29, 2023 08:08
Copy link
Collaborator

@alxmrs alxmrs left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry, this is a big enough PR that I can’t give a decent review while traveling. My main question is if the overhead for the thread pool will slow down this request vs just making the requests in a loop.

Maybe Fred or Nate can take a look here.

@dabhicusp
Copy link
Collaborator Author

dabhicusp commented Dec 1, 2023

Hello @alxmrs I tried with just making the requests in a loop. but it takes much more time(~21x) as compare to threadpool for the calculating in the image which user provided.

Code:

    tiles_threadpool = [None for _ in range(total_tile)]
    tiles_forloop = [None for _ in range(total_tile)]
    
    start_time = time.time()
    with concurrent.futures.ThreadPoolExecutor() as pool:
      for i, arr in pool.map(
          self._get_tile_from_EE, list(zip(data, cycle([coordinate_type])))
      ):
        tiles_threadpool[i] = (
            arr.tolist() if coordinate_type == 'longitude' else arr.tolist()[0]
        )
    print("threadpool_time: ", coordinate_type,time.time() - start_time)

    start_time = time.time()
    for i in range(total_tile):
      _, res = self._get_tile_from_EE((data[i], coordinate_type))
      tiles_forloop[i] = (res.tolist() if coordinate_type == 'longitude' else res.tolist()[0])
    print("for_loop_time : ", coordinate_type, time.time() - start_time)
    

Image which tested:

ee_class = EarthEngineBackendEntrypoint()
geom = ee.Geometry.Rectangle([-61, -7, -50, -18])
county = ee.Feature(geom)
image = ee.Image('projects/mapbiomas-workspace/TRANSVERSAIS/ZONACOSTEIRA6/mosaic_1985').clip(county)
coll = ee.ImageCollection(image)
ds = ee_class.open_dataset(coll.select(['NDVI']), crs='EPSG:3857',
                         scale=30, geometry=geom,
                         io_chunks={'time':24,'X': 512, 'Y': 512}
                         )

Result:

threadpool_time:  longitude 4.9227306842803955
for_loop_time :  longitude 95.86925983428955
threadpool_time:  latitude 4.389440298080444
for_loop_time :  latitude 74.4054172039032

Also tested on other images too and got the same result as above(taken ~21X more times).

Copy link
Collaborator

@naschmitz naschmitz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for tackling this!

xee/ext.py Outdated Show resolved Hide resolved
xee/ext.py Outdated Show resolved Hide resolved
xee/ext.py Outdated Show resolved Hide resolved
xee/ext.py Outdated Show resolved Hide resolved
xee/ext.py Outdated Show resolved Hide resolved
xee/ext.py Outdated Show resolved Hide resolved
xee/ext.py Outdated Show resolved Hide resolved
@naschmitz naschmitz self-requested a review December 19, 2023 15:31
@copybara-service copybara-service bot merged commit 284dd79 into google:main Dec 19, 2023
6 of 11 checks passed
@dabhicusp dabhicusp deleted the grid_dimension branch December 20, 2023 06:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Pixel grid dimensions (40818x1) must be less than or equal to 32768.
3 participants